Large language models for biomedicine: foundations, opportunities, challenges, and best practices.

OBJECTIVES: Generative large language models (LLMs) are a subset of transformers-based neural network architecture models. LLMs have successfully leveraged a combination of an increased number of parameters, improvements in computational efficiency, and large pre-training datasets to perform a wide spectrum of natural language processing (NLP) tasks. Using a few examples (few-shot) or no examples (zero-shot) for prompt-tuning has enabled LLMs to achieve state-of-the-art performance in a broad range of NLP applications. This article by the American Medical Informatics Association (AMIA) NLP Working Group characterizes the opportunities, challenges, and best practices for our community to leverage and advance the integration of LLMs in downstream NLP applications effectively. This can be accomplished through a variety of approaches, including augmented prompting, instruction prompt tuning, and reinforcement learning from human feedback (RLHF). TARGET AUDIENCE: Our focus is on making LLMs accessible to the broader biomedical informatics community, including clinicians and researchers who may be unfamiliar with NLP. Additionally, NLP practitioners may gain insight from the described best practices. SCOPE: We focus on 3 broad categories of NLP tasks, namely natural language understanding, natural language inferencing, and natural language generation. We review the emerging trends in prompt tuning, instruction fine-tuning, and evaluation metrics used for LLMs while drawing attention to several issues that impact biomedical NLP applications, including falsehoods in generated text (confabulation/hallucinations), toxicity, and dataset contamination leading to overfitting. We also review potential approaches to address some of these current challenges in LLMs, such as chain of thought prompting, and the phenomena of emergent capabilities observed in LLMs that can be leveraged to address complex NLP challenge in biomedical applications.

Investigating the Association Between Dynamic Driving Pressure and Mortality in COVID-19-Related Acute Respiratory Distress Syndrome: A Joint Modeling Approach Using Real-Time Continuously-Monitored Ventilation Data.

IMPORTANCE AND OBJECTIVES: COVID-19-related acute respiratory distress syndrome (ARDS) is associated with high mortality and often necessitates invasive mechanical ventilation (IMV). Previous studies on non-COVID-19 ARDS have shown driving pressure to be robustly associated with ICU mortality; however, those studies relied on "static" driving pressure measured periodically and manually. As "continuous" automatically monitored driving pressure is becoming increasingly available and reliable with more advanced mechanical ventilators, we aimed to examine the effect of this "dynamic" driving pressure in COVID-19 ARDS throughout the entire ventilation period. DESIGN SETTING AND PARTICIPANTS: This retrospective, observational study cohort study evaluates the association between driving pressure and ICU mortality in patients with concurrent COVID-19 and ARDS using multivariate joint modeling. The study cohort (n = 544) included all adult patients (≥ 18 yr) with COVID-19 ARDS between March 1, 2020, and April 30, 2021, on volume-control mode IMV for 12 hours or more in a Mass General Brigham, Boston, MA ICU. MEASUREMENTS AND MAIN RESULTS: Of 544 included patients, 171 (31.4%) died in the ICU. Increased dynamic ΔP was associated with increased risk in the hazard of ICU mortality (hazard ratio [HR] 1.035; 95% credible interval, 1.004-1.069) after adjusting for other relevant dynamic respiratory biomarkers. A significant increase in risk in the hazard of death was found for every hour of exposure to high intensities of driving pressure (≥ 15 cm H2O) (HR 1.002; 95% credible interval 1.001-1.003). CONCLUSIONS: Limiting patients' exposure to high intensities of driving pressure even while under lung-protective ventilation may represent a critical step in improving ICU survival in patients with COVID-19 ARDS. Time-series IMV data could be leveraged to enhance real-time monitoring and decision support to optimize ventilation strategies at the bedside.

Characterizing terminology applied by authors and database producers to informatics literature on consumer engagement with wearable devices.

OBJECTIVE: Identifying consumer health informatics (CHI) literature is challenging. To recommend strategies to improve discoverability, we aimed to characterize controlled vocabulary and author terminology applied to a subset of CHI literature on wearable technologies. MATERIALS AND METHODS: To retrieve articles from PubMed that addressed patient/consumer engagement with wearables, we developed a search strategy of textwords and Medical Subject Headings (MeSH). To refine our methodology, we used a random sample of 200 articles from 2016 to 2018. A descriptive analysis of articles (N = 2522) from 2019 identified 308 (12.2%) CHI-related articles, for which we characterized their assigned terminology. We visualized the 100 most frequent terms assigned to the articles from MeSH, author keywords, CINAHL, and Engineering Databases (Compendex and Inspec together). We assessed the overlap of CHI terms among sources and evaluated terms related to consumer engagement. RESULTS: The 308 articles were published in 181 journals, more in health journals (82%) than informatics (11%). Only 44% were indexed with the MeSH term "wearable electronic devices." Author keywords were common (91%) but rarely represented consumer engagement with device data, eg, self-monitoring (n = 12, 0.7%) or self-management (n = 9, 0.5%). Only 10 articles (3%) had terminology from all sources (authors, PubMed, CINAHL, Compendex, and Inspec). DISCUSSION: Our main finding was that consumer engagement was not well represented in health and engineering database thesauri. CONCLUSIONS: Authors of CHI studies should indicate consumer/patient engagement and the specific technology investigated in titles, abstracts, and author keywords to facilitate discovery by readers and expand vocabularies and indexing.

Scalable Feature Engineering from Electronic Free Text Notes to Supplement Confounding Adjustment of Claims-Based Pharmacoepidemiologic Studies.

Natural language processing (NLP) tools turn free-text notes (FTNs) from electronic health records (EHRs) into data features that can supplement confounding adjustment in pharmacoepidemiologic studies. However, current applications are difficult to scale. We used unsupervised NLP to generate high-dimensional feature spaces from FTNs to improve prediction of drug exposure and outcomes compared with claims-based analyses. We linked Medicare claims with EHR data to generate three cohort studies comparing different classes of medications on the risk of various clinical outcomes. We used "bag-of-words" to generate features for the top 20,000 most prevalent terms from FTNs. We compared machine learning (ML) prediction algorithms using different sets of candidate predictors: Set1 (39 researcher-specified variables), Set2 (Set1 + ML-selected claims codes), and Set3 (Set1 + ML-selected NLP-generated features), vs. Set4 (Set1 + 2 + 3). When modeling treatment choice, we observed a consistent pattern across the examples: ML models utilizing Set4 performed best followed by Set2, Set3, then Set1. When modeling the outcome risk, there was little to no improvement beyond models based on Set1. Supplementing claims data with NLP-generated features from free text notes improved prediction of prescribing choices but had little or no improvement on clinical risk prediction. These findings have implications for strategies to improve confounding using EHR data in pharmacoepidemiologic studies.

Learning from undercoded clinical records for automated International Classification of Diseases (ICD) coding.

OBJECTIVES: To develop an unbiased objective for learning automatic coding algorithms from clinical records annotated with only partial relevant International Classification of Diseases codes, as annotation noise in undercoded clinical records used as training data can mislead the learning process of deep neural networks. MATERIALS AND METHODS: We use Medical Information Mart for Intensive Care III as our dataset. We employ positive-unlabeled learning to achieve unbiased loss estimation, which is free of misleading training signal. We then utilize reweighting mechanism to compensate for the imbalance between positive and negative samples. To further close the performance gap caused by poor quality annotation, we integrate the supervision provided by the automatic annotation tool Medical Concept Annotation Toolkit which can ease the heavy burden of manual validation. RESULTS: Our benchmarking results show that positive-unlabeled learning with reweighting outperforms competitive baseline methods over a range of missing label ratios. Integrating supervision provided by annotation tool further boosted the performance. DISCUSSION: Considering the annotation noise and severe imbalance, unbiased loss estimation and reweighting mechanism are both important for learning from undercoded clinical records. Unbiased loss requires the estimation of false negative ratios and estimation through trained models is practical and competitive. CONCLUSIONS: The combination of positive-unlabeled learning with reweighting and supervision provided by the annotation tool is a promising solution to learn from undercoded clinical records.

Using an optimized generative model to infer the progression of complications in type 2 diabetes patients.

BACKGROUND: People live a long time in pre-diabetes/early diabetes without a formal diagnosis or management. Heterogeneity of progression coupled with deficiencies in electronic health records related to incomplete data, discrete events, and irregular event intervals make identification of pre-diabetes and critical points of diabetes progression challenging. METHODS: We utilized longitudinal electronic health records of 9298 patients with type 2 diabetes or prediabetes from 2005 to 2016 from a large regional healthcare delivery network in China. We optimized a generative Markov-Bayesian-based model to generate 5000 synthetic illness trajectories. The synthetic data were manually reviewed by endocrinologists. RESULTS: We build an optimized generative progression model for type 2 diabetes using anchor information to reduce the number of parameters learning in the third layer of the model from [Formula: see text] to [Formula: see text], where [Formula: see text] is the number of clinical findings, [Formula: see text] is the number of complications, [Formula: see text] is the number of anchors. Based on this model, we infer the relationships between progression stages, the onset of complication categories, and the associated diagnoses during the whole progression of type 2 diabetes using electronic health records. DISCUSSION: Our findings indicate that 55.3% of single complications and 31.8% of complication patterns could be predicted early and managed appropriately to potentially delay (as it is a progressive disease) or prevented (by lifestyle modifications that keep patient from developing/triggering diabetes in the first place). CONCLUSIONS: The full type 2 diabetes patient trajectories generated by the chronic disease progression model can counter a lack of real-world evidence of desired longitudinal timeframe while facilitating population health management.

Using Twitter data to understand public perceptions of approved versus off-label use for COVID-19-related medications.

OBJECTIVE: Understanding public discourse on emergency use of unproven therapeutics is essential to monitor safe use and combat misinformation. We developed a natural language processing-based pipeline to understand public perceptions of and stances on coronavirus disease 2019 (COVID-19)-related drugs on Twitter across time. METHODS: This retrospective study included 609 189 US-based tweets between January 29, 2020 and November 30, 2021 on 4 drugs that gained wide public attention during the COVID-19 pandemic: (1) Hydroxychloroquine and Ivermectin, drug therapies with anecdotal evidence; and (2) Molnupiravir and Remdesivir, FDA-approved treatment options for eligible patients. Time-trend analysis was used to understand the popularity and related events. Content and demographic analyses were conducted to explore potential rationales of people's stances on each drug. RESULTS: Time-trend analysis revealed that Hydroxychloroquine and Ivermectin received much more discussion than Molnupiravir and Remdesivir, particularly during COVID-19 surges. Hydroxychloroquine and Ivermectin were highly politicized, related to conspiracy theories, hearsay, celebrity effects, etc. The distribution of stance between the 2 major US political parties was significantly different (P < .001); Republicans were much more likely to support Hydroxychloroquine (+55%) and Ivermectin (+30%) than Democrats. People with healthcare backgrounds tended to oppose Hydroxychloroquine (+7%) more than the general population; in contrast, the general population was more likely to support Ivermectin (+14%). CONCLUSION: Our study found that social media users with have different perceptions and stances on off-label versus FDA-authorized drug use across different stages of COVID-19, indicating that health systems, regulatory agencies, and policymakers should design tailored strategies to monitor and reduce misinformation for promoting safe drug use. Our analysis pipeline and stance detection models are made public at https://github.com/ningkko/COVID-drug.

Identifying and Reconciling Patients' Allergy Information Within the Electronic Health Record.

Allergy information is often documented in diverse sections of the electronic health record (EHR). Systematically reconciling allergy information across the EHR is critical to improve the accuracy and completeness of patients' allergy lists and ensure patient safety. In this retrospective cohort study, we examined the prevalence of incompleteness, inaccuracy, and redundancy of allergy information for patients with a clinical encounter at any Mass General Brigham facility between January 1, 2018 and December 31, 2018. We identified 4 key places in the EHR containing reconcilable allergy information: 1) allergy modules (including free text comments and duplicate allergen entries), 2) medication laboratory tests results, 3) oral medication allergy challenge tests, and 4) medication orders that have been discontinued due to adverse drug reactions (ADRs). Within our cohort, 718,315 (45.2% of the total 1,588,979) patients had an active allergy entry; of which, 266,275 (37.1%) patient's records indicated a need for reconciliation.

The intersection of big data and epidemiology for epidemiologic research: The impact of the COVID-19 pandemic.

Big data epidemiology facilitates pandemic response by providing data-driven insights by utilizing big data tools that differ from traditional methods. Aspects regarding 'garbage in, garbage out', such as insufficient data, inaccessibility of data, missing data, uncertainty in handling data and bias in analysis or common findings are addressable by combining techniques across disciplines.

Embedding, aligning and reconstructing clinical notes to explore sepsis.

OBJECTIVE: Our goal was to research and develop exploratory analysis tools for clinical notes, which now are underrepresented to limit the diversity of data insights on medically relevant applications. RESULTS: We characterize how exploratory analysis can affect representation learning on clinical narratives and present several self-developed tools to explore sepsis. Our experiments focus on patients with sepsis in the MIMIC-III Clinical Database or in our institution's research patient data repository. We found that global embeddings assist in learning local representations of clinical notes. Second, aligning at any specific time facilitates the use of learning models by pooling more available clinical notes to form a training set. Furthermore, reconstruction of the timeline enhances downstream-processing techniques by emphasizing temporal expressions and temporal relationships in clinical documentation. We demonstrate that clustering helps plot various types of clinical notes against a scale, which conveys a sense of the range or spread of the data and is useful for understanding data correlations. Appropriate exploratory analysis tools provide keen insights into preprocessing clinical notes, thereby further enhancing downstream analysis capabilities, making data driven medicine possible. Our examples can help generate better data representation of clinical documentation for models with improved performance and interpretability.

Estimating Time to Progression of Chronic Obstructive Pulmonary Disease With Tolerance.

We defined tolerance range as the distance of observing similar disease conditions or functional status from the upper to the lower boundaries of a specified time interval. A tolerance range was identified for linear regression and support vector machines to optimize the improvement rate (defined as IR) on accuracy in predicting mortality risk in patients with chronic obstructive pulmonary disease using clinical notes. The corpus includes pulmonary, cardiology, and radiology reports of 15,500 patients who died between 2011 and 2017. Their performance was compared against a long short-term memory recurrent neural network. The results demonstrate an overall improvement by those basic machine learning approaches after considering an optimal tolerance range: the average IR of linear regression was 90.1% and the maximum IR of support vector machines was 66.2%. There was a similitude between the time segments produced by our tolerance algorithms and those produced by the long short-term memory.

Predicting Falls Among Community-Dwelling Older Adults: A Demonstration of Applied Machine Learning.

Data science skills are increasingly needed by informatics nurses and nurse scientists, but techniques such as machine learning can be daunting for those with clinical, rather than computer science or technical, backgrounds. With the increasing quantity of publicly available population-level datasets, identification of factors that predict clinical outcomes is possible using machine learning algorithms. This study demonstrates how to apply a machine learning approach to nursing-relevant questions, specifically an approach to predict falls among community-dwelling older adults, based on data from the 2014 Behavioral Risk Factor Surveillance System. A random forest algorithm, a common approach to machine learning, was compared to a logistic regression model. Explanations of how to interpret the models and their associated performance characteristics are included to serve as a tutorial to readers. Machine learning methods constitute an increasingly important approach for nursing as population-level data are increasingly being made available to the public.

Following data as it crosses borders during the COVID-19 pandemic.

Data change the game in terms of how we respond to pandemics. Global data on disease trajectories and the effectiveness and economic impact of different social distancing measures are essential to facilitate effective local responses to pandemics. COVID-19 data flowing across geographic borders are extremely useful to public health professionals for many purposes such as accelerating the pharmaceutical development pipeline, and for making vital decisions about intensive care unit rooms, where to build temporary hospitals, or where to boost supplies of personal protection equipment, ventilators, or diagnostic tests. Sharing data enables quicker dissemination and validation of pharmaceutical innovations, as well as improved knowledge of what prevention and mitigation measures work. Even if physical borders around the globe are closed, it is crucial that data continues to transparently flow across borders to enable a data economy to thrive, which will promote global public health through global cooperation and solidarity.

A temporal visualization of chronic obstructive pulmonary disease progression using deep learning and unstructured clinical notes.

BACKGROUND: Chronic obstructive pulmonary disease (COPD) is a progressive lung disease that is classified into stages based on disease severity. We aimed to characterize the time to progression prior to death in patients with COPD and to generate a temporal visualization that describes signs and symptoms during different stages of COPD progression. METHODS: We present a two-step approach for visualizing COPD progression at the level of unstructured clinical notes. We included 15,500 COPD patients who both received care within Partners Healthcare's network and died between 2011 and 2017. We first propose a four-layer deep learning model that utilizes a specially configured recurrent neural network to capture irregular time lapse segments. Using those irregular time lapse segments, we created a temporal visualization (the COPD atlas) to demonstrate COPD progression, which consisted of representative sentences at each time window prior to death based on a fraction of theme words produced by a latent Dirichlet allocation model. We evaluated our approach on an annotated corpus of COPD patients' unstructured pulmonary, radiology, and cardiology notes. RESULTS: Experiments compared to the baselines showed that our proposed approach improved interpretability as well as the accuracy of estimating COPD progression. CONCLUSIONS: Our experiments demonstrated that the proposed deep-learning approach to handling temporal variation in COPD progression is feasible and can be used to generate a graphical representation of disease progression using information extracted from clinical notes.

Rethinking Data Sharing at the Dawn of a Health Data Economy: A Viewpoint.

A health data economy has begun to form, but its rise has been tempered by the profound lack of sharing of both data and data products such as models, intermediate results, and annotated training corpora, and this severely limits the potential for triggering economic cluster effects. Economic cluster effects represent a means to elicit benefit from economies of scale from internal data innovations and are beneficial because they may mitigate challenges from external sources. Within institutions, data product sharing is needed to spark data entrepreneurship and data innovation, and cross-institutional sharing is also critical, especially for rare conditions.

Natural Language Processing and Its Implications for the Future of Medication Safety: A Narrative Review of Recent Advances and Challenges.

The safety of medication use has been a priority in the United States since the late 1930s. Recently, it has gained prominence due to the increasing amount of data suggesting that a large amount of patient harm is preventable and can be mitigated with effective risk strategies that have not been sufficiently adopted. Adverse events from medications are part of clinical practice, but the ability to identify a patient's risk and to minimize that risk must be a priority. The ability to identify adverse events has been a challenge due to limitations of available data sources, which are often free text. The use of natural language processing (NLP) may help to address these limitations. NLP is the artificial intelligence domain of computer science that uses computers to manipulate unstructured data (i.e., narrative text or speech data) in the context of a specific task. In this narrative review, we illustrate the fundamentals of NLP and discuss NLP's application to medication safety in four data sources: electronic health records, Internet-based data, published literature, and reporting systems. Given the magnitude of available data from these sources, a growing area is the use of computer algorithms to help automatically detect associations between medications and adverse effects. The main benefit of NLP is in the time savings associated with automation of various medication safety tasks such as the medication reconciliation process facilitated by computers, as well as the potential for near-real-time identification of adverse events for postmarketing surveillance such as those posted on social media that would otherwise go unanalyzed. NLP is limited by a lack of data sharing between health care organizations due to insufficient interoperability capabilities, inhibiting large-scale adverse event monitoring across populations. We anticipate that future work in this area will focus on the integration of data sources from different domains to improve the ability to identify potential adverse events more quickly and to improve clinical decision support with regard to a patient's estimated risk for specific adverse events at the time of medication prescription or review.

A value set for documenting adverse reactions in electronic health records.

Objective: To develop a comprehensive value set for documenting and encoding adverse reactions in the allergy module of an electronic health record. Materials and Methods: We analyzed 2 471 004 adverse reactions stored in Partners Healthcare's Enterprise-wide Allergy Repository (PEAR) of 2.7 million patients. Using the Medical Text Extraction, Reasoning, and Mapping System, we processed both structured and free-text reaction entries and mapped them to Systematized Nomenclature of Medicine - Clinical Terms. We calculated the frequencies of reaction concepts, including rare, severe, and hypersensitivity reactions. We compared PEAR concepts to a Federal Health Information Modeling and Standards value set and University of Nebraska Medical Center data, and then created an integrated value set. Results: We identified 787 reaction concepts in PEAR. Frequently reported reactions included: rash (14.0%), hives (8.2%), gastrointestinal irritation (5.5%), itching (3.2%), and anaphylaxis (2.5%). We identified an additional 320 concepts from Federal Health Information Modeling and Standards and the University of Nebraska Medical Center to resolve gaps due to missing and partial matches when comparing these external resources to PEAR. This yielded 1106 concepts in our final integrated value set. The presence of rare, severe, and hypersensitivity reactions was limited in both external datasets. Hypersensitivity reactions represented roughly 20% of the reactions within our data. Discussion: We developed a value set for encoding adverse reactions using a large dataset from one health system, enriched by reactions from 2 large external resources. This integrated value set includes clinically important severe and hypersensitivity reactions. Conclusion: This work contributes a value set, harmonized with existing data, to improve the consistency and accuracy of reaction documentation in electronic health records, providing the necessary building blocks for more intelligent clinical decision support for allergies and adverse reactions.

Solving Interoperability in Translational Health. Perspectives of Students from the International Partnership in Health Informatics Education (IPHIE) 2016 Master Class.

BACKGROUND: In the summer of 2016 an international group of biomedical and health informatics faculty and graduate students gathered for the 16th meeting of the International Partnership in Health Informatics Education (IPHIE) masterclass at the University of Utah campus in Salt Lake City, Utah. This international biomedical and health informatics workshop was created to share knowledge and explore issues in biomedical health informatics (BHI). OBJECTIVE: The goal of this paper is to summarize the discussions of biomedical and health informatics graduate students who were asked to define interoperability, and make critical observations to gather insight on how to improve biomedical education. METHODS: Students were assigned to one of four groups and asked to define interoperability and explore potential solutions to current problems of interoperability in health care. RESULTS: We summarize here the student reports on the importance and possible solutions to the "interoperability problem" in biomedical informatics. Reports are provided from each of the four groups of highly qualified graduate students from leading BHI programs in the US, Europe and Asia. CONCLUSION: International workshops such as IPHIE provide a unique opportunity for graduate student learning and knowledge sharing. BHI faculty are encouraged to incorporate into their curriculum opportunities to exercise and strengthen student critical thinking to prepare our students for solving health informatics problems in the future.

Prevalence of food allergies and intolerances documented in electronic health records.

BACKGROUND: Food allergy prevalence is reported to be increasing, but epidemiological data using patients' electronic health records (EHRs) remain sparse. OBJECTIVE: We sought to determine the prevalence of food allergy and intolerance documented in the EHR allergy module. METHODS: Using allergy data from a large health care organization's EHR between 2000 and 2013, we determined the prevalence of food allergy and intolerance by sex, racial/ethnic group, and allergen group. We examined the prevalence of reactions that were potentially IgE-mediated and anaphylactic. Data were validated using radioallergosorbent test and ImmunoCAP results, when available, for patients with reported peanut allergy. RESULTS: Among 2.7 million patients, we identified 97,482 patients (3.6%) with 1 or more food allergies or intolerances (mean, 1.4 ± 0.1). The prevalence of food allergy and intolerance was higher in females (4.2% vs 2.9%; P < .001) and Asians (4.3% vs 3.6%; P < .001). The most common food allergen groups were shellfish (0.9%), fruit or vegetable (0.7%), dairy (0.5%), and peanut (0.5%). Of the 103,659 identified reactions to foods, 48.1% were potentially IgE-mediated (affecting 50.8% of food allergy or intolerance patients) and 15.9% were anaphylactic. About 20% of patients with reported peanut allergy had a radioallergosorbent test/ImmunoCAP performed, of which 57.3% had an IgE level of grade 3 or higher. CONCLUSIONS: Our findings are consistent with previously validated methods for studying food allergy, suggesting that the EHR's allergy module has the potential to be used for clinical and epidemiological research. The spectrum of severity observed with food allergy highlights the critical need for more allergy evaluations.